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Reasons for Allowance 

1. The following is an examiner's statement of reasons for allowance of claims 1 and 5 in 
the amendment of 2/15/2006: Nothing in the prior art anticipates or suggests, "a setting step, of 
automatically selecting and setting one mapping mode out of a plurality of mapping modes based 
on image placement information of the horizontal and vertical placement direction of the 
plurality of images having a common subject region" in an image synthesis method comprising 
an input step of inputting a plurality of image data representing a plurality of images; a 
placement information generating step of generating placement information about horizontal and 
vertical placement direction of the plurality of images, determined by the horizontal and vertical 
placement direction of the plurality of images represented by the image data inputted in the input 
step; a placement information obtaining step of obtaining the placement information about a 
plurality of images in which adjacent images have a common subject region; a setting step of 
automatically selecting and setting one mapping mode output of a plurality of mapping modes 
based on image placement information of the horizontal and vertical placement direction of the 
plurality of images having a common subject region, each mapping mode corresponding to a 
different mapping surface, without a user intervening to determine the mapping mode or the 
corresponding mapping surface; a synthesis step of combining the plurality of images by using 
the mapping mode set in the setting step; a changing step, of changing the mapping mode; and a 
generating step, of issuing when an image formed by changing the mapping mode in the 
changing step does not comply with a predetermined condition set in accordance with the 
mapping mode, a warning and generating and displaying a synthesized image in accordance with 
the predetermined condition, the warning being issued in a case in which the synthesized image 
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exceeds a predetermined angle of view while simultaneously generating and displaying the 
synthesized image within the predetermined range of viewing angle. 
2. The following is an examiner's statement of reasons for allowance of claim 8 in the 
amendment of 2/15/2006: Nothing in the prior art anticipates or suggests, "setting means for 
automatically selecting and setting one mapping mode out of a plurality of mapping modes based 
on image placement information of the horizontal and vertical placement direction of the 
plurality of images having a common subject region" in an image synthesis apparatus 
comprising: inputting means for inputting a plurality if image data representing a plurality of 
images; placement information generation means for generating placement information about 
horizontal and vertical placement direction of the plurality of images, determined by the 
horizontal and vertical placement direction of the plurality of images represented by the image 
data inputted by the inputting means; placement information obtaining means for obtaining the 
placement information about a plurality of images in which adjacent images have a common 
subject region; setting means for automatically selecting and setting one mapping mode out of a 
plurality of mapping modes based on image placement information of the horizontal and vertical 
placement direction of the plurality of images having a common subject region, each mapping 
mode corresponding to a different mapping surface, without a user intervening to determine the 
mapping mode or the corresponding mapping surface; synthesis means for combining the 
plurality of images by using the mapping mode set by the setting means; changing means for 
changing the mapping mode; generating means for issuing, when an image formed by changing 
the mapping mode by the changing means does not comply with a predetermined condition set in 
accordance with the mapping mode, a warning and generating and displaying a synthesized 
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image in accordance with the predetermined condition, the warning being issued in a case in 
which the synthesized image exceeds a predetermined angle of view while simultaneously 
generating and displaying the synthesized image within the predetermined range of viewing 
angle; and display means for displaying a cuttable rectangular region without a margin in the 
synthesized image. 

3. The following is an examiner's statement of reasons for allowance of claim 11 in the 
amendment of 2/15/2006: Nothing in the prior art anticipates or suggests, " a setting step of 
automatically selecting and setting one mapping mode out of a plurality of mapping modes based 
on image placement information of the horizontal and yertical placement direction of the 
plurality of images haying a common subject region " in a computer-generated storage medium 
having recorded thereon a program for implementing a computer-implementable image synthesis 
method for combining a plurality of images, the program comprising: an input step, of inputting 
a plurality of image data representing a plurality of images; a placement information generating 
step of generating placement information about horizontal and vertical placement direction of the 
plurality of images, determined by the horizontal and vertical placement direction of the plurality 
of images represented by the image data inputted in the input step; a placement information 
obtaining step of obtaining the placement information about a plurality of images in which 
adjacent images have a common subject region; a setting step of automatically selecting and 
setting one mapping mode out of a plurality of mapping modes based on image placement 
information of the horizontal and vertical placement direction of the plurality of images having a 
common subject region, each mapping mode corresponding to a different mapping surface, 
without a user intervening to determine the mapping mode or the corresponding mapping 



Application/Control Number: 09/624,385 Page 5 

Art Unit: 2628 

surface; a synthesis step of combining the plurality of images by using the mapping mode set in 
the setting step; a changing step of changing the mapping mode; and a generating step of issuing 
when an image formed by changing the mapping mode in the changing step does not comply 
with the a predetermined condition set in accordance with the mapping mode, a warning and 
generating and displaying a synthesized image in accordance with the predetermined condition, 
the warning being issued in a case in which the synthesized image exceeds a predetermined 
angle of view while simultaneously generating and displaying the synthesized image within the 
predetermined range of viewing angle. 

The cited reference, Teo U.S. Patent No. 6,246,413, does not teach automatically 
selecting and setting the mapping mode. Xiong et al. U.S. Patent No. 6,434,265 teaches in 
figures 2 and 3 a user interface and a global optimization that automatically provides feedback to 
the computer system when the pair- wise objective function is not desirable for a poor selection 
of the projection viewing plane and the resulting panoramas will have imperfectly aligned 
images that give shadow or ghosting effects due to a poor selection of the projection viewing 
plane and the resulting panoramas will have imperfectly aligned images that give shadow or 
ghosting effects. Xiong further teaches an alternative human interaction being present throughout 
the image synthesis process to provide feedback to the computer system in all the nonlinear 
optimizations to let users monitor the progress of the system and allow them to intervene when 
necessary . Xiong discloses that a panorama is constructed on a particular geometry that will best 
facilitate the rendering of the projection onto a chosen viewing plane (a geometric surface) such 
as cubic, polyhedral, cylindrical and spherical geometries. Xiong describes that a user can 
alternatively intervene to select a viewing plane. Xiong discloses that upon the user's intervening 
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of selecting a viewing plane, the computer program automatically selecting and setting the 
mapping mode in accordance to the parameters set by the user's intervening action. Teo teaches 
that the viewing software can automatically invert the projection by automatically straightening 
out the portion of the panoramic image being displayed (column 7-8). Teo teaches in column 10 
the mapping mode of mapping onto the planar surfaces such as mapping the panorama images 
onto a cube and the mapping mode of mapping the panorama images onto polyhedral surfaces to 
form cylindrical panorama images. Once the geometry is selected either by a user or by the 
viewer software , the image processor automatically transforms the first digital panoramic image 
into a second digital panoramic image corresponding to the projection of the scene onto the 
surface geometry. Therefore, the mapping mode is automatically selected and set by the viewer 
software because the steps involved a mode in the mapping or a mode in transforming the first 
digital panoramic image into a second digital panoramic image corresponding to the projection 
of the scene onto the changed/selected geometric surface are automatically performed by the 
software. Although Teo teaches that a user can change a mapping surface ( column 1 L lines 1*9) 
by adjusting the shape of the mapping surfaces so that a given cylindrical panoramic image can 
be transformed to produce a modified panoramic image, the procedure can be automated by the 
computer software. Teo therefore teaches automatically selecting and setting a mapping mode or 
changing the plane for mapping the panoramic image by the computer through a user interface or 
by the computer software without the user interface. Teo further discloses the horizontal 
placement of image by projecting a scene onto a cube so that horizontal lines of the scene appear 
horizontal in the panoramic image, and vertical placement of image by projecting the panoramic 
image on the vertical cylinder or by rotating the polyhedral surface relative to a cylindrical 
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panoramic image and making adjustments to the sides and edges of the polyhedral surface so that 
the placement of images appear vertically on the mapping surface and thus selecting and setting 
a mapping mode with the mapping surface being adjusted without a user intervention. Teo 
teaches the user interface enabling the user to rotate the polyhedral surface relative to a 
cylindrical panoramic image and making adjustments to the sides and edges of the polyhedral 
surface so that the placement of images appear vertically on the mapping surface. However, 
neither Teo or Xiong discloses the claim limitation of " a setting step of automatically selecting 
and setting one mapping mode out of a plurality of mapping modes based on image placement 
information of the horizontal and yertical placement direction of the plurality of images haying a 
common subject region "set forth in the independent claim 1, 8 and 11. 

Shum et al. U.S. Patent No. 6,271,855 teaches a plurality of mapping modes such as 
zooming and panning the screen images of the interior scene shown in the panoramic view of 
Fig. 10A wherein the mapping surfaces within panoramas are different and changing the type of 
panoramas such as the spherical panoramas and the planar panoramas. Shum teaches a setting 
step of zooming/panning the panorama and setting the spherical panoramas by the interactive 
user interface so that the mapping mode changes in which each resulting panorama has different 
mapping surfaces. Shum further teaches determining the constraints governing the mapping 
mode of the mapping onto the projection plane wherein the constraints incorporates the 
predetermined panoramic view of a 3D scene or the predetermined camera orientation matrix by 
the governing linear systems of equations. Shum further teaches generating a warning message to 
indicate that the constraint equations are unsolvable in a case in which the synthesized image is 
not aligning with any portion of the projected model in response to the change in the geometric 
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constraints determined by the projection plane or in response to the change in the panoramic 
view. The geometric constraints change due to the change in the projection plane from a planar 
panorama to a spherical panorama wherein the planes within the spherical panorama are changed 
to spherical planes and the panorama view can be changed as well by zooming/panning 
panorama and/or selecting a portion of the panorama. Therefore, un-solvability of the linear 
equations occurs in the process of changing the mapping mode because the change in the 
mapping mode or the change in the geometric surface also changes the geometric constraints 
governing the linear equations. When the linear systems of equations are not solvable, a user can 
switch the mapping plane orientation or changing the mode of projection surface to meet the 
constraint equations requirements wherein the plane normals have changed. Moreover, the 
projection space can be in the form of any shape such as spherical or planar and by changing the 
camera's point of view and/or the panorama view and the coordinates and directions of the plane 
normals, the projection space is effectively changed and accordingly the mapping mode is 
changed. 

However, Shum teaches changing the mapping mode by zooming and panning the screen 
images of the interior scene shown in the initial panoramic view of Fig. 10A (See Figs. 1 1 A and 
1 IB and column 23-24) and changing the type of panoramas such as the spherical panoramas 
(Figs. 15A-15B) and the planar panoramas (Figs. 16A-16B). Shum teaches a setting step of 
zooming/panning the panorama and setting the spherical panoramas (column 23-24) by the 
interactive user interface so that the mapping mode changes wherein each resulting panorama 
has different mapping surfaces in accordance with the user input commands (column 23-24) 
wherein the panorama mapping surfaces change as a result of the zooming and panning. 
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However, Shum does not disclose the claim limitation of " a setting step of automatically 
selecting and setting one mapping mode out of a plurality of mapping modes based on image 
placement information of the horizontal and vertical placement direction of the plurality of 
images having a common subject region "set forth in the independent claim 1, 8 and 11. 

4. Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany the issue 
fee. Such submissions should be clearly labeled "Comments on Statement of Reasons for 
Allowance." 



Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (571) 272-7665. 
The examiner can normally be reached on 8:00 - 6:30 (Mon-Thu). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee Tung can be reached on (571) 272-7794. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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